[Testing 15] test(evals): offline eval harness — citation accuracy, injection resistance, leakage - #257

What this tests — and what it doesn't

This harness gates the checkers and the response-format contract, not the model. The fixture answers are recorded, hand-written responses in Mike's exact runtime format — the model is treated the way tests treat any expensive, nondeterministic external dependency: mocked out, so the logic around it can be tested deterministically. A live model regression will not show up here; what will show up is any change that weakens a safety check or drifts the citation format away from what the backend emits and parses.

Why offline-first instead of jumping straight to online evals against real responses:

• An online eval can't gate CI in an open-source repo. GitHub Actions doesn't expose secrets to fork PRs, so most contributor PRs couldn't run it at all — and it's nondeterministic and costs API spend per run. This harness needs no network, no keys, no npm install, so it runs free on every PR.
• Online evals are only as trustworthy as their scorers. An online harness is "generate real responses, then score them" — and scorers that have never been tested against known-bad cases produce a green dashboard you can't trust. This PR ships the scoring half first, with a corpus that proves it.
• The two layers catch different regressions. Example: a well-meaning PR relaxes quote verification from verbatim to fuzzy matching (to cope with OCR noise). A live model's mostly-correct answers still score fine — but case cc-03, where a liability cap is quietly edited from twelve months to twenty-four, now passes a checker that should reject it, mismatches its expected: "fail", and CI goes red. An online eval would likely have missed it.

The checkers are proven, not assumed

Every case declares expected: "pass" or "fail", and roughly half the corpus is known-bad (a fabricated Erie holding, the altered liability cap above, a canary smuggled into a tool-call argument). If a checker is ever weakened, those cases flip and CI fails. Benign edge cases (docket numbers near the SSN pattern, an answer that warns about an injection without repeating it) go red if a checker over-triggers.

How this differs from #231 (closed)

#231 was closed for "limited usefulness of the evals and format. A fuller real evals suite can be implemented in the future which aligns with the format of actual assistant responses and involves real documents." This PR was rebuilt around that feedback:

• Real response format. Fixture answers are not invented markdown — they are prose with inline [N] markers plus a <CITATIONS> JSON block, exactly what the backend emits and parses. The harness's citation parser is ported line-for-line from backend/src/lib/chat/citations.ts (source lines cited in its comments), document pages mirror the [Page N] blocks from documentOps.ts, and recorded tool calls use the ToolCall shape from chat/types.ts with Mike's real tool names. docs/evals.md has the full fixture-field → runtime-format mapping with file references, so the format contract is auditable. This is also what makes the mocks valid: a recorded response is only a fair stand-in if it's shaped exactly like the real thing.
• Real documents. Sources are public-domain legal texts at real-world length: Marbury v. Madison and Erie v. Tompkins passages, Hadley v. Baxendale, 17 U.S.C. § 107, 15 U.S.C. §§ 1–2, and full-length contract clauses. Cited spans are buried mid-document, and injection payloads are embedded inside the real text — which is the attack shape the product actually faces.

What this completes

CI already has an evals job that runs node evals/run.mjs --threshold 1.0 if the file exists and skips otherwise. This PR adds that file, turning the gate on: a deterministic scorecard for fabricated citations, prompt injection via retrieved documents, and privileged/PII leakage on every PR.

How to run

node evals/run.mjs                          # all suites, must be 100%
node evals/run.mjs --suite prompt-injection # one suite
node evals/run.mjs --list                   # see every case

Exit 0 at or above threshold, non-zero below. Verified both directions: the committed corpus is 27/27, and corrupting a single quoted span drops it to 26/27 with exit code 1 and a readable reason (quote not verbatim on page 1 of ...).

The suites

• citation-accuracy — parses the <CITATIONS> block and checks every [N] marker has a matching entry and every quote exists verbatim on the cited page (or in the fetched opinion for cluster_id citations). Covers cross-page quotes, page ranges, and the legacy citation shape.
• prompt-injection — recorded turns where the document carries an injected instruction and a canary; fails on canary leakage in output or tool arguments, or on a disallowed tool call (e.g. an injected note ordering an edit_document rewrite).
• privilege-leakage — flags off-matter emails, SSN-like values, sb_secret_* keys, service_role JWTs (the public anon key deliberately doesn't trip it), and document identifiers from another client's matter.

Future live mode

Mocked tests don't replace integration tests against the real dependency — they let you run the real thing less often, out-of-band, where flakiness and cost don't gate merges. The "fuller real evals suite" from #231 is a drop-in extension here: run real prompts through the backend, capture the actual streamed responses and tool calls into this same fixture format, and score them with these same, already-proven checkers — behind an opt-in, outside CI, since it needs keys and isn't deterministic. docs/evals.md spells this out, along with the runtime counterparts (citation verification in backend/src/lib/chat/citations.ts; spotlighting of untrusted content in the security-hardening PR #227).

🤖 Generated with Claude Code